A driving need in analyzing and improving software operation and performance is for measures such as Software Reliability to mature the "physics of failure" and design aspects related to software reliability. This type of focus would then enhance one's ability to effect reliable software in a predictable form. A major challenge is that software reliability, in essence, requires one to measure compliance to customer/user requirements. Customer/user requirements can range over a wide spectrum of software product attributes that relate directly or indirectly to software performance. Identifying and measuring these attributes in a structured way to minimize risk and allow pro-active preventive action during software development is no easy task. The Goal-Question-Metric paradigm, discussed by Basili see V. R. Basili, "Software Modeling and Measurement: The Goal/Question/Metric Paradigm", University of Maryland at College Park Computer Science Technical Report UMIACS-TR-92-96, 1992 Sep, pp 1-24 (herein referred to as "Basili")!, is one popular and effective approach to measurement identification. However, in practice, additional challenges in using this approach have been encountered. Some of these challenges, though, seem to be alleviated with use of a reliability technique called success/fault tree analysis. Experience has shown that the Goal-Question-Metric paradigm is conducive to the building of G-Q-M trees which may be analyzed using reliability success/fault tree logic.
A number of working sessions were held to identify an initial, standard set of software metrics for use across our projects. Participants included software team leaders, software managers, software engineering process group members and software quality assurance staff. It was decided that the software measures should be goal-driven to ensure that there was both an "owner" of the metric and a method of determining if, and when, the metric had outlived it's usefulness. Basili's Goal/Question/Metric paradigm was used as a process guide within the working sessions. However, during facilitation of these sessions, a number of obstacles were encountered. These obstacles included the following:
1) Participants had different definitions of reliability and quality; PA0 2) Participants had different organizational perspectives on the same goal, question or metric; PA0 3) Participants wanted a way to quantitatively assess, overall, how they were progressing when faced with a large number of product attributes related to quality and reliability; PA0 4) Participants were not current on the literature related to definitions of reliability metrics and relationships of metrics; PA0 5) Participants were quite often short on patience in brainstorming compatible Goals, Questions and Metrics that fit their situation; PA0 6) Participants quickly saw complexities such as goal tradeoffs, competing and conflicting goals, and the need for a dynamic model to handle changing Goal/Question/Metric desires; PA0 7) Participants realized that different Questions and Metrics, related to reliability and quality, were sometime heavily influenced depending on the data already collected by the projects; PA0 8) Participants desired the ability to select Goal/Question/Metric groupings that would minimize the total number of data primitives to be collected; and PA0 9) To save time and frustration, participants desired a strawman of a Goal/Question/Metric taxonomy to aid in preparation prior to each working session. PA0 1) Support for project planning and control; PA0 2) Determination of strengths and weaknesses; PA0 3) A rationale for the adoption and refinement of various software engineering techniques or methods; PA0 4) Permits assessment of the impact of changes in techniques and methods; and PA0 5) Support for evaluation of both software processes and products.
From this experience, several lessons-learned were apparent. First, the need to become more knowledgeable of the detailed workings of the Goal/Question/Metric paradigm see Basili!. Second, the need to identify a goal flow-down methodology within the organization so that the paradigm could be applied within organizational levels. Third, the need for a detailed process for determining how to originate questions for specific goals, and how to originate metrics for specific questions. Fourth, the need for a way to make the Goal/Question/Metric result actionable in a variety of situations.
The Goal/Question/Metric paradigm see Basili! is directed at the development of a set of corporate, division and project goals related to different business measures such as customer satisfaction, quality, financial progress, technical performance, etc. The paradigm includes: 1) a process of identifying goals followed by 2) the generation of questions that help define the goals which is then followed by specifying 3) specific measures to answer the questions. Inherent in the paradigm is the additional focus on developing mechanisms for data collection, validation and analysis, and taking action on the results.
Organizations implementing this paradigm in the development of their measurement program gain lift in many ways:
However, an important benefit of implementing this paradigm is that it keeps the focus on why the metrics are being collected.
In practice, goals are specified at an organizational level and then tailored and integrated into the organizational process and product models - based on the needs of the organization. In this manner, goals may be traced to data and data may be interpreted to understand goal performance. Templates are identified in Basili to provide guidance in the formulation of the goals, questions and metrics.
Goals were identified to possess five attributes: 1) the object of interest, 2) the purpose of studying the object of interest, 3) the focus with regard to characteristics of the object of interest, 4) who's perspective is to be supported by the goal, and 5) within which context or environment the object is to be studied.
After goal setting, a set of questions are identified for each goal to provide corroboration, clarity, and coverage of the different aspects of the goal. After question development, multiple metrics are identified for each question in the same manner. See FIG. 1 for an example of a Goal/Question/Metric tree related to Software Reliability using a known definition of software reliability see J. A. McCall, "Specifications of Software Quality Attributes, Vols I, II, III", U.S. Rome Air Development Center Final Technical Report RADC-TR-85-37, 1985 Feb (herein referred to as "McCall")!.
By following the above paradigm, the resulting measurement framework is goal-driven and maintainable via feedback from the "owners" of the goals, questions and metrics.